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ABSTRACT 

This study of survey research experts was conducted 
to determine desirable characteristics of mail questionnaires. The 
82--item Likert-scale instrument used in the study covered general 
appearance, instructions, choice of items, choice of response 
options, wording, order of items, and item format. The instrument was 
administered to: 8 subjects who participated in a study in which the 
instrument was developed, and 12 other experts (10 of whom functioned 
as a validation panel). For the eight early participants, reasons 
were elicited regarding circumstances under which they would use 
certain types of items that they indicated would only be used 
"sometimes". Frequency distributions were prepared for all items for 
the original group of participants and for the validation panel, 
kesults indicate that while there are some mail questionnaire 
recommendations that could usually be made with some degree of 
confidence, there are other aspects of questionnaire design that are 
less commonly accepted and, hence, dependent upon researcher 
knowledge and experience, information sought, population 
characteristics, and circumstances c The survey questionnaire used is 
included. (TJH) 
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Consensus on Desirable Characteristics of Mail Questionnaires: 

Illusion or Reality? 



Objectives 

The purpose of the present study was to determine if agreement could be found among survey 
research experts regarding desirable characteristics of mail questionnaires when they rated those 
characteristics as ones they would recommend usually, some of the time, or seldom. For items 
that would not usually be recommended, there was interest in deteraiining the cucumstances under 
which they would be applicable. A second purpose was to determine if a second group of survey 
researchers (termed the "vaUdation group" in this paper) would support the findings of the group 
of experts who might conceivably have been biased by their participation in a previous similar 
study. 

Background 

Recent research directed toward finding a set of desirable characteristics of mail questionnaires 
that would be endorsed by a group of survey research experts did not obtain its objective (Clark & 
Boser, 1989). A list of 83 characteristics of mail questionnaires was compiled from published 
sources on survey research. In a prior .tudy this list was submitted to a sample of experts for 
rating, but only eight of the 83 items found total acceptance. Some items lacked clarity, as 
indicated by participant comments. There were two items that none of the participants would 
recommend for all mail questionnaires. 

Comments emanating from the first study had led the researchers to the conclusion that the 
poles of the initial response categories ("all" and "none") may have been too extreme, causing 
participants to reject thiem for the only other available option, "some" ("recommended for some but 
not all mail surveys") if they thought of one or more applications for which the recommendation 
would or would not be made. Limitations of the instrument were thus considered as a possible 
factor contributing to the lack of consensus among the experts. This concern led to revisions of the 
instrument. The response option from the original study of "all" ("recommended for all mail 
survey questionnaires") was broadened to "usuaUy=usuaUy or always recommended" while the 
other extreme, originally stated as "non.i" ("not recommended"), was revised to become 
"seldom=seldom or never recommended." The middle category of "some" ("recommended for 
some but not all mail questionnaires") was changed only to reflect the tone and r -^sing of the new 
response options, becoming "sometimes=sometimes recommended for mail su. . 
questionnaires." For the original participants, each section of the revised questionnaire was 
followed by a request for the participant to clarify responses of "sometimes" or "seldom" by 
delineating the circumstances under which that item would be applicable. 
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Almost aU of the items from the original (Clark & Boser, 19?9) study were used in the cuirent 
study. Four of the five items which had caused confusion to participants in the previous study 
were rewritten in an effort to clarify them, and the fifth such item was deleted. 

Method 

Subjects 

EhasfiJ.. The 1 1 participants in the previous study (Clark & Boser, 1989) were given an 
opportunity for continued involvement in the project. It was recognized that there was a possibility 
that participation in die first phase of the study and knowledge of the results of die first study might 
bias responses of members of the original study group, so a second sample, the validation sample, 
was selected. 

Ehassl. A group size of 10 was deemed desirable for the validation panel. Oversampling was 
initiated by sending questionnaires to 15 individuals to obtain 10 participants for the vaUdation 
panel. The 15 individuals had backgrounds in survey research, as did those in die original survey, 
but had not participated in the first study. 
Instrument 

Tne revised instrument contained 82 items in seven categories: general appea^^nce (14 items), 
instructions (8 items), choice of items (7 items), choice of response options (10 items), wording 
(12 items), order of items (15 items), and item format ( 16 items). For die onginal participants 
(Phase 1), following each section of items (a section contained items of one of the seven types), 
participants were asked to indicate die circumstances or types of surveys in which items rated as 
"sometimes" would be recommended. If there was not sufficient blank space at die bottom of the 
page, the facing page was left blank for this purpose. Participants in Phase 2, die validation study, 
were not asked to explain or list circumstances relevant to items they had rated as "sometimes." 

Questionnaires for Phase 1 participants were duplicated on blue legal size paper diat was 
stapled in the middle to form a booklet. Because of die additional space required for explanatory 
comments, the questionnaire had 12 pages. Questionnaires for Piiase 2 participants were green 
and required only eight pages because die spaces for explanations were not needed. 
Participants 

The respondents in Phase 1 consisted of eight individuals, all of whom bad participated in the 
previous research effort (Clark & Boser, 1989) and had eidier published looks on survey 
methodology or reported on studies investigating survey research methodology at national 
professional conferences. Two other participants indicated diey thought their responses would be 
biased by dieir participation in die initial study and declined to participate. One of diem 
recommended a colleague to participate in his place, and die odier forwarded his questionnaire to a 
colleague. Bodi recommended individuals became part of die validation group. Only one of die 
original 1 1 participants failed to respond in any manner. 
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The respondents in the validation group (Phase 2) consisted of the first 10 individuals (from 
the 15 sent questionnaires) who returned completed questionnaires. In all, a total of 12 from the 
validation sample returned the survey instruments without benefit of follow-ups. Background 
information was provided by nine of the 10 validation panel members." 

For the validation group, experience in survey research activities varied from five to 30 years, 
with a median of 13 years. Participants had been involved in conducting from none (n=l) to eight 
(n=l) mail surveys in 1988, with a mean of three. Five of the nine participants indicated 1988 was 
typical of their survey activities, but the remaining four indicated they were usually more active in 
survey research. Eight of the nine had conducted and reported studies of survey or questionnaire 
methodology, and four had published articles or books on survey methodology. Eight of the 10 
individuals were involved in institutional or organizational research, and five of them limited their 
activities to this type. Other types of survey research included public opinion (n=2), consumer 
(n=l) and social science (n=l). Populations surveyed included members of organizations (n=5), 
program participants (n=5), alumni (n=4), and the geneial public (n=3). Seven of the 10 weie 
college/university faculty members, with three employed by research ins. .utes within their 
institutions. Two individuals were employed by research divisions or sections in large 
organizations, and one individual v/as in a public school research division. 
Procedures 

Survey forms, cover letters and postage-paid return envelopes (along with previously promised 
copies of the results of the first sun'ey) were mailed to the original 1 1 participants in April. The 
cover letter emphasized the change in response options. One follow-up mailing, containing a 
letter, a replacement copy of the questionnaire and a stamped, addre.<:sed return envelope, was sent 
approximately one month later. Completed survey instruments were received from eight of the 1 1 
original participants (73 percent). 

Survey forms, cover letters and postage-paid return envelopes were mailed to the 15 
individuals selected for the validation sample in mid-June. Responses from 12 of them constituted 
an 80 percent response rate. 
Analysis 

Frequency distributions were prepared for all items for the original group participants in Phase 
1 and for the validation panel in Phase 2. Explanatory comments listing special circumstances in 
which practices would sometimes be recommended were noted for items on which there was not 
total agreement by participants in the original group (Phase 1). 

Items were listed in four groups, based on responses of the original group: items on which al.' 
who responded marked the item as "usually" recommended, items on which all but one who 
responded marked the item as "usually" recommended, items on which all but two who responded 
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marked the item as "usuaUy" recommended, and the items that lacked general acceptance as usual 
practices. 

After items were grouped according to responses by the original participants, the percentage of 
validation panel members who would "usually" recommend each item was calculated. If an item 
v/as "usually" recommended by 80 percent or more of the validation panel, the item was 
considered to have been supported. 

Results and Discussion 

Frequency distributions for e ich item for the original participants and the validation panel are 
appended. On 26 of the 82 items (32 percent), all eight of the original participants agreed that they 
would "usually" recommend the item for mail survey questionnaires. And on another 8 items, aU 
of those responding to tiie item (n=6 or n=7) rated the items as usually recommended. Those items 
are listed below. 

On 25 of the 34 items, the validation panel agreed at the 80 percent level or higher. Those 
items appear in bold-face tjije in the Ust that follows. Those items not in bold-face type were 
supported by fewer than 80 percent of the validation panel. 



Items Recommended by All of Original Group 



A. GENERAL APPEARANCE 

1 . The title of the study/questionnaire is likely to appeal to the survey population. 

2. Instrument looks easy to complete. 
6. Type is clear and legible. 

11. Appreciation for complet'ng the instrument is expressed. 
9a. The front page (or cover) contains the study/instrument tiUe, prominently displayed. 

B. INSTRUCTIONS 

2. Instructions are brief. 

3a. Instructions are clear: They specify when to put a check mark and when to 
write m a response. 

3b. Instructions are clear: They indicate whether multiple responses are allowed. 

6. If items appear on both sides of the page, an indication is given that the instrument continues on the 
reverse side (e.g., "please turn over"). 

C. CHOICE OF ITEMS 

2. Each item seeks just one piece of information. 

3. All items are essential and relevant to the purposes of the survey. 

4b. For items used for skip/filter/screen purposes, instructions are few and simple. 
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D. CHOICE OF RESPONSE OPTIONS 

la. Response options exhanst all possibiHties or include "other," "iiDdecJded " or 
"neutral" category* 

Id. Response options do not contain more than one alternative that could be correct 

unless multiple responses are allowed. 
Ig. Response options are appropriate for the item. 

1 f . Response options are brief. 

2b. Items with Likert-type response options use a balanced scale. (n=7) 

E. WORDING 

1. The choice of words is appropriate to the literacy level of the Sw^vey population. 
3d. Items are simple, direct, and unambiguous. They do not contain instances of 

double negatives in items and/or response options. (n=7) 
3e. Items are simple, direct, and unambiguous. They do not contain instances of 

negatively worded items coupled with agree/disagree response format. (n=7) 

F. ORDER OF ITEMS 

lb. The initial items are applicable to all members of the survey population. 
Id. The initial items ak-e nonthreatening. 
li The initial items are interesting. 

5. If reference is made to a previous item, that item appears on the sanze page or on 
the facing page* *^ * 

6. Items with similar content are grouped together; within each content group, items 
with the same response format are presented together* 

la. The initial items are clearly connected to the slated purpose of the survey. 
4. Classification or demographic information is solicited at the end of the instrument unless needed for 
screening purposes. 

7b. Within a topic/content area, the items progress from most familiar to least familiar. 

7c. Withm a topic/content area, the items progress from least objectionable to most objectionable. 

G. ITEM FORMAT 

8. Response options are arranged vertically (or in columns if several consecutive 
items use the same response options). (n=6) 

9. Response options are close to tht- item stem. (n=7) 
11. There is adequate space for responding. (n=7) 

13. When ranking, the number of items to be ranked is limited (e.g., three best and 
three worst). (n=5) 

10. The space for responding to items is on the same side of the page throughout the instalment. (n=6) 
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On another 18 items, only one of the original participants rated the item as "sometimes'* or 
"seldom," whUe the rest accorded it the "usually" rating. Comments or special circumstances 
from those original participants arc shown in italics to the right of the item. 

Using the same 80 percent agreement, 12 of these items were supported by the validation 
sample and are again shown in bold-face type. 



Items Usually Recommended by All But One of Original Group 



Item 



A3. 

AS. 

A8. 

B5. 
CL 



C4c. 

Dlb. 

E3b. 

E3c. 

E3g. 



Margins are adequate; instrument doesn^t 
look crowded. 

Printing does not bleed through the paper. 

There are not too many variations in size and style of type. 



The tone of the directions is polite (e.g., "please")* 

The respondent is able to provide answers to the 
questions in the instrument. 



For items used for skip/filter/screen purposes, instructions 
appear immediately after the response opdons. (n=6) 

Response options are mutually exclusive. 



Items are simple, direct, and unambiguous. 
They dO not contain instances of "loaded" items 
(that use emotionally colored words). (n=6) 

Items are simple, direct, and unambiguous. They do not 
contain instances of assumption of an existing slate of affairs 
(e.g., ••Do you still..."). 

Items are simple, direct, and unambiguous. They 
do not contain instances of giveaway" words 
(e.g., "air'). 



Flc. The initial items are easy. (n=6) 



F7d. Within a topic/content area, the items progress 
from objective to subjective. 



Circumstances/Comments 

Crowding preferable to longer 
instrumenu 



Interesting, varied format may add 
appeal for children, teens, or 
specialized audiences. 



Person should have the knowledge; 
sometimes opinions of naive 
persons sought (although t!m dont 
feel able to provide the answers). 



Except items with" check all 
that apply," 

Such words are appropriate in 
attitude surveys. 



MOTE: Wording of this item 
was confusing to some participants. 



NOTE: Wording of this item 
was confusing to some participants. 



More important that initial items 
be interesting if there are no items 
that are both interesting and easy. 

No clear agreement on this. 
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Items Usually Recommended by AH But One of Original Group (continued) 



Item 



Circumstances/Comments 



F8» Items that require reca\I are organized by logical 
time sequence. (n=6) 

If necessary, either sublettering (e.g.. 4a. 4b. 4c) or numbering 
by sections (i.e.. starting each section with item 1) is used to 
limit the apparent number of items. 

Each item and its response options are on 
the same page. 

Statements or questions, rather than phrases, are used in 
collecting demographic information (e.j.. "How old were 
you on your last birthday?" instead of "Age."). (n=6) 

G14b. For checklists^ column headings are carried over 
from one page to another* (n=6) 

Gl4c. For checklists, column headings are presented parallel, rather 
than perpendicular, to the item stem. (n=5) 



No clear agreement on this. 



G2. 



G3. 



G4. 



Long items (25 Likert items) 
may not fit on a page. 

Level of detail needed and 
literacy of reader must be 
considered. 

Better not to have to carry over 
to second page. 

There may be too many options 
at times. 



For the following 12 items, all but tNvo of the original gioup agreed that ihey should be 
recommended. Only one of the items, G7, was validated at the 80 percent level. 



Items Usually Recommended by AH But Two of Original Group 



Item 



Circumstances/Comments 



Size and style o 'type used for headings is consistent 
throughout the instrument. Consistency is also evident for 
items and response options. 

The front page (or cover) contains general directions. 

Instructions aie visually different from the body of the 
instrument (e.g., in size and/or style of type). 

Die. Response options include both sides of issue or question. 



A7. 

A9b. 
B4. 



E2. 

E3f. 
E3i. 



F7a. 

Gl. 
G5. 

G7. 



Both sides of issue (or neither side) are inducted in the 
item stem. 



Items are simple, direct, and unambiguous. They do not 
contain instances of qualifying clauses, especially at end of stem. 

Items are simple, direct, and unambiguous. They do not 
contain instances of vague terminology (e.g.. "the country," 
"just." "fair." "you"). 



Within a topic/content area, the items progress from 
general to specific. 



Items are numbered with Arabic numerals, (n =5 ) 

If an item stem requires two or more lines, the second 
and subsequent lines arc indented. (n=5) 

When response options are provided (includL..g, 
if appropriate, a response option of "other"), each 
response option has either a numeric or alphabetic 
code beside it. (n=5) 



Preferably in cover letter. 



Beth or neithp.r: some issues may 
have more than two viewpoints 

There may be more than two sides. 
Some items may focus on one side 
of an issue. 

NOTE. Wording of this item was 
confusing to some participants. 

Many of the terms acceptable for 
response categories. This statement 
doesn't specify what part of the 
item the words are to be omitted 
from. 

Sometimes may not care if earlier 
responses influence summary item. 
No clear agreement. 



012. Open-ended items are used sparingly. (n=5) 



Recommended to facilitate com- 
puter data entry. Otherwise 
checking beside or circling 
response may be acceptable. 

If most likely responses are known. 
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The foUov^ing items appear to be more controversial or highly situational, or the items in this 
study were not clearly written. These items would not be included among a generic list of 
questionnaire characteristics. Some of the circumstances are Hsted. 



Non-Generic, or Situation-Specific Items 



Item 



A4. Paper is white or light-colored with dark ink. 



A9c. The front page (or cover) contains the narne of the sponsor. 



A9d The front page (or cover) contains the address of the sponsor. 



A 10. For a multi-page questionnaire, the back page does not 
contain items but may be used for comments. 



Bl. 

B3c. 

C4a. 



General instructions that apply to the entire instrument 
aro provided at the beginning of the instrument 

Instructions are clear. They provide guidance for expected 
length of open-ended responses. 

For items used for skip/filter/screen purposes, the use of this 
type is justified. 



C4d. For items used for skip/filter/screen purposes items pertaining 
to only some of the respondents are indented beneath the 
filter question. 

Die. Response options include a "don't know" option. 



Circumstances/Comments 



Brighter colors, varied formats may 
be more appealing to children, 
teens, specific audiences. 

Some prefer this in cover letter. 
Response rate may be lower if very 
personallcoffidential information 
sought. 

Some prefer this in cover letter or 
at end of questionnaire. 
Complete address not needed in 
organization survey. 

Use for questions if needed rather 

than adding pages. 

Less important if comments hMve 

been sought throughout the 

questionnaire. 



ff there is a need to lima length. 
Available space is an indicator. 

One justification is to shorten the 
overall questionnaire. 
Avoid when possible in mail 
surveys. 

This format may be less appro- 
priate for map-like or flowchart- 
based formats. 

Some prefer to force respondents 
to make a choice. 
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Item 



Non-Generic, or Situation-Specific Items (continued) 



Circumstances/Comments 



D2a. Items with Likerl-type response options have an appropriately 
labeled midpoint 



Some prefer not to include a 
midpoint or to label only the 
end points. 



D3. Sensitive information (e.q., age. salary) is collected using 
ranges for response options 



E3a. Items are simple, direct, and unambiguous. They do not 
c(Milain instances of jargon, technical terms, or uncommon 
abbreviations. 



E3h. Items are simple, direct, and unambiguous. They do not 
contain instances of inexact words or phrases (e.g., "any." 
"most," "several," "usually," "often." "regularly," "much the 
same"). 

E3j. Items are simple, direct, and unambiguous. They do not 
contain instances of the word "questionnaire" or "checklist" 
in heading or text 

F2. Tf there are any sensitive or difficult items, they appear in 
the middle or near the end of the instrument, but not at the 
very end. 



F3. Open-ended items appear last 



< '6, The respondent is asked to circle or underline responses. 



Gl4a. For checklists, if long, a line is skipped after every three to 
six items. 



Unless interval level data needed. 
Ranges may be preferable if 
anonymity or confidentiality is 
a concern. 

Technical terms can be used if those 
in the sample would be familiar 
with them (engineering terms for 
a survey of engineers). 

These are acceptable response 
categories, and this item doesn't 
state whether these words are to be 
omitted from the stem or response. 



All item:> may be sensitive. 
Place at end unless critical to 
study (more commitment to 
answer because of time already 
spent on the questionnaire). 

Should follow items if used to 
clarify or expand responses to them. 

Except when listing responses 
might influence respondents 
or when possible responses 
cannot be predicted. 

May not be needed if items go 
across most of the page or if 
there is sufficient space between 
items. 



There was much less agreement on the use of items in section G than in other sections. There 
was also more reluctance to rate the items in that section, possibly indicating confusion over the 
items themselves. 
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Summary 

There were 34 items on which all (or all who rated the item) of the original panel agreed that 
they would usually recommend. For 25 of those items, there was 80 percent or higher supoort 
from the validation panel that the items should be included in a list of recommendations usually 
made in mail surveys. Of the 18 items on which all but one of the original panel supported, 12 of 
the validation panel provided support And on the 12 items on which all but two of the original 
panel would usually recommend, only one of the items was supported by the validation panel. In 
summary, of the 64 items which a majority of the original participants would usually recommend, 
38 of them were supported by die validation group at the 80 percent or higher level mdicating they 
also would usually make the recommendation. As consensus declined within the original 
participants, the support of the validation group also declined. 

There are 18 items from the instrument that appear to be recommendations tiiat would be made 
only in certain circumstances or were poorly written and confusing to participants. Li a few cases, 
participants mdicated they would b? more likely not to make such a recommendation than to make 
it. 

The method of analysis for these data was arbitrary. The level-of-agreement criterion for the 
original participants (i.e., all but two or fewer agreeing the recommendation would usually be 
made) and the 80 percent criterion for the validation panel may be too lenient The comments from 
the original sample regarding items that are situation-specific or confusing in themselves may lead 
to improvement of the wording or intent of some items and the acceptance that thei« are conditions 
under which others are applicable. 

Based on tiiis and the previous research, it appears that while there are some mail questionnaire 
recommendations that could usually be made with some degree of confidence, there are other 
aspects of questionnaire design thai are less commonly accepted, and their proper use may depend 
on tne experience and knowledge of the researcher regarding not only questionnaire design but also 
the population to be surveyed, the information sought, and the circumstances. In other words, 
questionnaire design may be a science only up to a certain pouit; beyond that point it is an m, and • 
it would appear that point is reached somewhere prior to questionnaire design completion. 

Reference 

Clark, S. B., & Boser, J. A. (1989, March). Seeking consp.n^ms or, ^mpirj^^i ^hp^>^^typc;nY^ 
effgctivg mail questionnaires: A first step. Paper presented at the annual meeting of the 
American Educational Research Association, San Francisco. (ERIC Document Reprxxiuction 
Service No. ED 306 305) 
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Percentage Distribution of Responses for Original-Participant and Validation Groups 



- — Orlglna: Participant Group (N=8) — 
Item Code Usually S ometimes Seldom NqRi»^, 



General Appearance 



Al 


100.0% 


0.0% 


0.0% 


0 0% 


A2 


100.0% 


0.0% 


o!o% 


0.0% 


A3 


87.5% 


12.5% 


0.0% 


0.0% 


A4 


62.5% 


37.5% 


0.0% 


0.0% 


AS 


87.5% 


12.5% 


0.0% 


0.0% 


A6 


100.0% 


0.0% 


0.0% 


0.0% 


A7 


75.0% 


25.0% 


0.0% 


0.0% 


AS 


87.5% 


12.5% 


0.0% 


0.0% 


A9a 


100.0% 


0.0% 


0.0% 


0.0% 


A9b 


75.0% 


12.5% 


12.5% 


0.0% 


A9g 


50.0% 


37.5% 


12.5% 


0.0% 


A9d 


12.5% 


75.0% 


115% 


0.0% 


AlO 


25.0% 


50.0% 


25.0% 


0.0% 


All 


100.0% 


0.0% 


0.0% 


0.0% 



Instructions 
Bl 
B2 
B3a 
B3b 
B3c 
B4 
B5 
B6 



62.5% 
100.0% 
100.0% 
100.0% 
37.5% 
75.0% 
87.5% 
100.0% 



Choice of Items 



CI 
C2 
C3 
C4a 
C4b 
C4c 
C4d 



87.5% 
100.0% 
100.0% 

62.5% 
100.0% 

75.0% 

62.5% 



12.5% 
0.0% 
0.0% 
0.0% 
37.5% 
25.0% 
0.0% 
0.0% 



12.5% 
0.0% 
0.0% 

12.5% 
0.0% 

12.5% 

37.5% 



Choice of Response Options 



Dla 
Dlb 
Die 
Did 
Die 
Dlf 
Dig 
D2a 
D2b 
D3 



100.0% 
87.5% 
25 0% 

100.0% 
75.0% 

100.0% 

100.0% 
37.5% 
87.5% 
37.5% 



0.0% 
12.5% 
75.0% 

0.0% 
25.0% 

0.0% 

0.0% 
50.0% 

0.0% 
62.5% 



25.0% 
0.0% 
0.0% 
0.0% 

25.0% 
0.0% 

12i>% 
0.0% 



0.0% 
0.0% 
0.0% 
25.0% 
0.0% 
0.0% 
0.0% 



0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
12.5% 
0.0% 
0.0% 



Validation Group (N=10) 

Usually Sometimes Seldom Nn Resn. 





10.0% 


0.0% 


10.0% 


on nc!?, 


10.0% 


0.0% 


0.0% 


90.0% 


10.0% 


0.0% 


0.0% 


40.0% 


50.0% 


10.0% 


0,0% 


80.0% 


10.0% 


10.0% 


0.0% 


90.0% 


10.0% 


0.0% 


0,0% 


60.0% 


30.0% 


0.0% 


10,0% 


60.0% 


30.0% 


0.0% 


10.0% 


70.0% 


20.0% 


10.0% 


0.0% 


70.0% 


20.0% 


10.0% 


0.0% 


60.0% 


20.0% 


20,0% 


0,0% 


30.0% 


50.0% 


20.0% 


0,0% 


10.0% 


40.0% 


40.0% 


10.0% 


80.0% 


10.0% 


0.0% 


10.0% 



0.0% 


60.0% 


30,0% 


10.0% 


0.0% 


0,0% 


80.0% 


20.0% 


0.0% 


0,0% 


0.0% 


80.0% 


20.0% 


0,0% 


0.0% 


0,0% 


100,0% 


0.0% 


0.0% 


0.0% 


0.0% 


40.0% 


50.0% 


10.0% 


0.0% 


0.0% 


40.0% 


60.0% 


0.0% 


0.0% 


0.0% 


80,0% 


20,0% 


0.0% 


0.0% 


0,0% 


70,0% 


30.0% 


0.0% 


0.0% 


0.0% 


80,0% 


10.0% 


0.0% 


10.0% 


0.0% 


90.0% 


10,0% 


0.0% 


0.0% 


0.0% 


90.0% 


10.0% 


0.0% 


0.0% 


0.0% 


60.0% 


10,0% 


10.0% 


20,0% 


0.0% 


90.0% 


10.0% 


0.0% 


0,0% 


123% 


60,0% 


30,0% 


10.0% 


0.0% 


0.0% 


60.0% 


30.0% 


10.0% 


0.0% 


0.0% 


80.0% 


10.0% 


10.0% 


0.0% 


0.0% 


90.0% 


10.0% 


0.0% 


0,0% 


0.0% 


20.0% 


70.0% 


10,0% 


0.0% 


0,0% 


90,0% 


10,0% 


0.0% 


0.0% 


0.0% 


60.0% 


30.0% 


10.0% 


0.0% 


0.0% 


70.0% 


30.0% 


0.0% 


0.0% 


0.0% 


100.0% 


0.0% 


0.0% 


0.0% 


0.0% 


50.0% 


30.0% 


20.0% 


0.0% 


12.5% 


70.0% 


30.0% 


0,0% 


0.0% 


0.0% 


60.0% 


40.0% 


0.0% 


0.0% 
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Percentage Distribution of Responses for Original-Pmicipant and Validation Groups 



— Original Participant Group (N=8).. 



•YaHdation Group (N=10)-- 



Item Code 


Usually Sometimes 


Seldom 


No ResD. 


Usuallv Sometimes 


Wording 














El 


100.0% 


0.0% 


0.0% 


0.0% 




U.UvO 


E2 


62.5% 


25.0% 


0.0% 


123% 


40.0% 


60.0% 


E3a 


62.5% 


373% 


0.0% 


0.0% 


100.0% 


0.0% 


E3D 


75.0% 


12.5% 


0.0% 


123% 


on r\QL 


1 A r\nt 
iU.Uvo 


E3c 


873% 


123% 


0.0% 


0.0% 


70 0% 


JU.U /O 




873% 


0.0% 


0.0% 


123% 


100 0% 


0 OQ(i 


E3e 


873% 


0.0% 


0.0% 


123% 


so 0% 


70 


E3f 


75.0% 


25.0% 


0.0% 


0.0% 


70 0% 




E3g 


873% 


123% 


0.0% 


0.0% 


1000% 


\J,\J /o 


E3h 


373% 


623% 


0.0% 


0.0% 


400% 




E3i 


75.0% 


25.0% 


0.0% 


0.0% 


70.0% 


10 0% 


E3j 


373% 


50.0% 


123% 


0.0% 


40.0% 


40 0% 


Order of Hems 












Fla 


100.0% 


0.0% 


0.0% 


0.0% 


60 0% 

\J\J%\J /u 




Fib 


100.0% 


0.0% 


0.0% 


0.0% 


80.0% 


20 0% 


Flc 


75.0% 


123% 


0.0% 


123% 


90.0% 




Fid 


100.0% 


0.0% 


0.0% 


0.0% 


90.0% 


10 0% 


Fie 


100.0% 


0.0% 


0.0% 


0.0% 


90.0% 


10.0% 


F2 


50.0% 


37.5% 


123% 


0.0% 


50.0% 


50.0% 


F3 


50.0% 


373% 


123% 


0.0% 






F4 


100.0% 


0.0% 


0.0% 


0.0% 


60.0% 


'^O 0% 


F5 


100.0% 


0.0% 


0.0% 


0.0% 


80.0% 


20.0% 


F6 


100.0% 


0.0% 


0.0% 


0.0% 


90.0% 


10.0% 


F7a 


75.0% 


25.0% 


0.0% 


0.0% 


70.0% 


30.0% 


F7b 


100.0% 


0.0% 


0.0% 


0.0% 


70.0% 




F7c 


100.0% 


0.0% 


0.0% 


0.0% 


50.0% 


50.0% 


F7d 


873% 


123% 


0.0% 


0.0% 


80.0% 


20.0% 


F8 


75.0% 


12 5% 


0.0% 


123% 


90.0% 


10.0% 


Item Format 














Gl 


62.5% 


25.0% 


0.0% 


123% 


50.0% 


40.0% 


G2 


75.0% 


123% 


0.0% 


123% 


40.0% 


30.0% 


03 


75.0% 


123% 


0.0% 


123% 


90.0% 


0.0% 


G4 


75.0% 


123% 


0.0% 


123% 


30.0% 


50.0% 


G5 


623% 


12.5% 


123% 


123% 


40.0% 


30.0% 


06 


50.0% 


373% 


0.0% 


123% 


90.0% 


10.0% 


G7 


623% 


25.0% 


0.0% 


123% 


80.0% 


20 0% 


G8 


75.0% 


0.0% 


0.0% 


25.0% 


8o!o% 


20.0% 


G9 


873% 


0.0% 


0.0% 


123% 


90.0% 


10.0% 


GIO 


75.0% 


0.0% 


0.0% 


25.0% 


70.0% 


30.0% 


Gil 


873% 


0.0% 


0.0% 


123% 


90.0% 


10.0% 


G12 


62.5% 


25.0% 


0.0% 


123% 


70.0% 


30.0% 


G13 


623% 


0.0% 


0.0% 


373% 


80.0% 


10.0% 


G14a 


50.0% 


37.5% 


0.0% 


123% 


60.0% 


20.0% 


G14b 


75.0% 


123% 


0.0% 


123% 


100.0% 


0.0% 


G14c 


623% 


123% 


0.0% 


25.0% 


40.0% 


40.0% 


OVERALL 


77.4% 


153% 


2.6% 


4.4% 


70.6% 


233% 



0.0% 

0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
C.0% 
0.0% 
0.0% 
20.0% 



0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
10.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 



10.0% 
20.0% 
0.0% 
20.0% 
30.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
)0.0% 
0.0% 
0.0% 



4.1% 



0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 



0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 



0.0% 
10.0% 
10.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
0.0% 
10.0% 
10.0% 
0.0% 
20.0% 



1.7% 
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DESIRABLE CHARACTERISTICS 
OF MAIL QUESTIONNAIRES 



li^I . ^J ^ J -^^^"-^i '.^^ following pages are some generally agreed upon 
XZnZ2Td '^T"' """^ ^'^P'-'onnaires. Please indicate the relative importLce of each 
fh^fdlZigb ■ "^'^ l^^^i^onnaires by circling your response to the right of the item on 



cowi^fi;.^'" '^^"'^''^ °'' recommended for mail survey questionnaires 

c ~ ''ecommendedfor mail survey questionnaires 

SELDOM = seldom or never recommended for mail survey questionnaires 

Comments may be added at the bottom of pages on which items appear. 



JUDITH A. BOSER 
The University of Tennessee 

SHELDON B. CLARK 
Oak Ridge Associated Universities 



NOTE: QUESTIONNAIRE PAGE WAS 7" BY 8^5" FOR ACTUAL INSTRUMENT. 
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Please indtcaU the relative importance of each characteristic for mail survey question- 
naxres byc'rcling your response to the right of the item on the following basis: 
USUALLY = usually or al}y ffy^ recommended 
SOMETIMES a sometime^ recommended 

SELDOM « seldom cr n^vf r recommended 



A. General A ppearance 

1 . The title of the study/questionnaire is likely to appeal 

to the survey population USUALLY SOMEHMES 

1 Instrument looks easy to complete USUALLY SOMETIMES 

3. Margins arc adco wtc; instrument doesn't look crowded USUALLY SOMETIMES 

4. Paper is white or h-ht-colored with dark ink USUALLY SOMETIMES 

5. Printing docs not bleed through paper. USUALLY SOMETIMES 

6. Type is clear and legible USUALLY SOMETIMES 

7. Size and style of type xas/A for headings is consistent 
throughout the instrume .t Consistency is also evident 

for items and response, options USUALLY SOMETIMES 

8- There are not too mary variations in size and style of type USUALLY SOMETIMES 

9. The front page (or cover) contains: 

a. the studyAnstrument title, prominently displayed USUALLY SOMETIMES 

b. general directions USUALLY SOMETIMES 

c. the name of the sponsor. USUALLY SOMETIMES 

d. the address of the sponsor. USUALLY SOMETIMES 

10. For a multi-page questionnaire, the back page docs 

not contain items but may be used for comments USUALLY SOMETIMES 

1 1. Appreciation for completing the instrument is expressed USUAI.LY SOMETIMES 



SELDOM 
SELDOM 
SELDOM 
SELDOM 
SELDOM 
SELDOM 

SELDOM 
.^CLDOM 

SELDOM 
SELDOM 
SELDOM 
SELDOM 

SELDOM 
SELDOM 
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Please indicate the relative importance cf each chat acteristic for maU survzy question- 
naires by circling your » -sponse to the right of the item on the following basis: 
USUALLY - usuj.! !" ^ rQ*r^ recommended 
SOMETIMES a sometimifs recommended 
SELDOM seldom or never recommended 



B. Instruction's 

1 . General instructions thai apply to the entire instrument arc 

provided at the beginning of the instrumcnL USUALLY 

2. Instructions are brief. USUALLY 

3. Instructions are clean 

t. They specify when to put a check mark and when to 

write in a response USUALLY 

b They indicate whether multiple responses are allowed USUALLY 

c. They provide guidance for expected length of open- 

ended responses USUALLY 

4. Instniclions are visually different faoin the body of the 

instrument (e.g., in size and/or style of type) USUALLY 

5. The tone of the directions is polite (e.g., '*pl^e*') USUALLY 

6. If items appear on both sides of the page, an indication 
is given hat die instrument continues on the reverse side 

(e.g.. "^please turn over") JSUALLY 



SOMETIMES 
SOMETIMES 



SOME'HMES 
SOMETIMES 



SOMETIMES 
SOMETIME.^ 



SELDOM 
SELDOM 

SELDOM 
SELDOM 



SOMETIMES SELDOM 



SELDOM 
SEI^DOM 



SOMETIMES SELDOM 



Cnoice of Items 

1 . ^ The respondent is able to provide answers to the 

^quesdons in the instrument USUALLY SOMETIMES 

2. Each item seeks just one piece of information USUALLY SOMETIMES 

3.. All items arc essential and relevant to the purposes 

of the survey USUALLY 

4. For items used for skip/filtcr/scrccn purposes: 

a. The use of this type is justified USUALLY SOMETIMES 

b. Instructions are few and simple USUALLY SOMETIMES 

c. Instructions appear immediately after the 
response options USUALLY SOMETIMES 

d. Items pertaining to only sonw of tiie respondents arc 
indented beneath the filter question USUALLY SOMETIMES 



SELDOM 
SELDOM 

SOMETIMES SELDOM 

SELDOM 
SELDOM 

SELDOM 

SELDOM 
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Please indicate the relative importance of each characteristic for mail survey question- 
rMires by circling your response to the right of the item on the following basis: 
USUALLY =s usually or a/w^^j recommended 
SOMETIMES « som^iimtl recommended 
SELDOM a seldom or n^.vp.r recommended 



Choice of Response Optiny^^j 
Respons-* options: 

a. exhaust all possibilities or include "other," 

"undecided," or "neut^-al" category USUALLY 

b. ait mutually exclusive USUALLY 

c. include a "don't know" option USUALLY 

<i do not contain more than one alternative that could 

be correct unless multiple responses are allowed USUALLY 

e. include both sides of issue in question USUALLY 

f. are brief. USUALLY 

g. are appropriate for the item USUALLY 

Items with Likcrt-typc response options: 

a, have an appropriately labeled midpoint USUALLY 

b. use a balanced scale USUALLY 

SensiUve information (e.g., age, salary) is collected using 

ranges for response options USUALLY 



SOMETIMES 
SOMETIMES 
SOMETIMES 

SOMETIMES 
SOMETIMES 
SOMETIMES 
SOMETIMES 

SOMETIMES 
SOMETIMES 



SELDOM 
SELDOM 
SELDOM 

SELDOM 
SELDOM 
SELDOM 
SELDOM 

SELDOM 
SELDOM 



SOMETIMES SELDOM 
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Please indicate the relative importance cf each characteristic for mail survey question- 
naires by circling your respaise to the right of the item on the fcllo^ving basis: 
USUALLY » usually or alwgy^ recommended 
SOMETIMES a sometimes recommended 

SELDOM =» seldom or n^v^r r^mmm^M^^^ 



E. Wording 

1 . The choice of words is appropriate to the literacy level 

of the survey population USUALLY 

2. Both sides of an issue (or neither side) are included in the 

item stem USUALLY 

3. items arc simple, direct, and unambiguous. Thev do not 
contain instances of any of the- following pitfalls: 

a. jargon, technical terms, or uncommon abbreviations USUALLY 

b. loaded" items (that use emotionally colored wonk) USUALLY 

c. assumption of an existing state of affairs 

(e.g., "Do you sdU../) USUALLY 

d. douWe negatives in items and/or response options USUALLY 

c. negatively wooded items coupled with agicc/disagrcc 

response format USUALLY 

f. qualifying clauses, especially at end of stem USUALLY 

g. "giveaway" words (e.g., "all") USUALLY 

h. inexact words or phrases (e.g., "any," "most,** "jcveral," 

"usually," "often," "regularly," "much Uie same"). USUALLY 

i. vague terminology (e.g., "the country," "just," "fair," 

USUALLY 

j. the word "questionnaire" or "checklist" in heading 

^^^^ USUALLY 



SOMETIMES 
SOMETIMES 



SOMETIMES 
SOMETIMES 

SOMETIMES 
SOMETiMES 

SOMETIMES 
SOMETIMES 
SOMETIMES 



SELDOM 
S2LD0M 



SELDOM 
SELDOM 

SELDOM 
SELDOM 

SELDOM 
SELDOM 
SELDOM 



SOMETIMES SELDOM 



SOMETIMES 
SOMETIMES 



SELDOM 
SELDOM 
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Pleast indicase the relative importance cfeach characUrisiic for mail ,tuyey Question^ 
nmres ^ (^^rcUng your response to the right of the item on the following basis: 
USUALLY = lit w/v or /w^/r... j,.cc-^.n£ndcd 
SOMETIMES = sometime^ recommended 
SELDOM » seldom nr n^v^r recommended 



b. 
c. 
d 
e. 



F. Order of TtPm^ 

1. The initial items are: 

a. cleariy connected to the stated purpose of the survey. USUALLY 

applicable tc aii members of the survey population USUALLY 

USUALLY 

nonlhrealening USUALLY 

^"^^^"^"8 USUALLY 

2. If there are any sensitive or difficult items, they q)pcar 
in the middle or near the end of the instrument, but not 

at the very end USUALLY 

3. Open-ended items appear last. USUALLY 

4. aassification or demographic information is solicited at the 

end of the instrument unless needed for screening purposes USUALLY 

5. If reference is made to a previous item, that item appears 

on the same page or on the facing page USUALLY 

6. Items with similar content are grouped together, within each 
content group, items with the same response fomial are 

presented together USUALLY 

7. Within a topic/content area, the items progress from: 

a. general to specific USUALLY 

b. most familiar to least familiar. USUALLY 

c. least objectionable to most objectionable USUALLY 

d. objective to subjective USUA^JLY 

1. Items that require recall are organized by logical time 
sequence 



SO^'HTI^IES 
SOME-HMES 
SOMETIMES 
SOMETIMES 
SOMETIMES 

SOMETIMES 
SOMETIMES 

SOMETIMES 

SOMETIMES 



SOMETIMES 
SOMETIMES 
SOMETIMES 
SOMETIMES 



SELDOM 
SELDOM 
SELDOM 
SELDOM 
SELDOM 

SELDOM 
SELDOM 

SELDOM 

SELDOM 



SOMETIMES SELDOM 



SELDOM 
SELDOM 
SELDOM 
SELDOM 



USUALLY SOMETIMES SELDOM 



ERLC 
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Please indicate the relative importance of each characteristic for maU survey question- 



by circling your response to the right of the item on the following basis: 
USUALLY =« usua lly or always recommended 
SOMETIMES ; 
SELDOM = 



somettmgf recommended 

: recommended 



G* Item Format 

1. Items are numbered with Arabic numerals USUALLY 

2. If necessary, either subletteiing (e.g., 4a. 4b. 4c) or 
numbering by sections (i.e.. starling each section 
with item 1) is used to limit the apparent number 

^'^^ USUALLY 

3. Each item and its response options are on the same page USUALLY 

4. State icnls or questions, rather than phrases, arc used 
in cdlecling demographic information (eg., "How old 

were you on your last birthday?" instead of "Age") USUALLY 

5. If an item stem requires two or more lines, the second 

and subsequent lines arc indented USUALLY 

6. The respondent is asJccd to circle or undcriine responses 

already presented rather than write them on a blank. USUALLY 

7. When response options arc provided (including, if appropriate, 
a response option of "other-), each response option has 

either a numeric or alphabetic code beside it USUALLY 

8. Response options are ananged vertically (or in columns 

if several onseculive items use the same response options) USUALLY 

9. Response options are close to the item stem. USUALLY 

1 0. The space for responding to items is on the same side 

of the page throughout the instrument USUALLY 

11. There is adequate space for responding USUALLY 

12. Open-ended items arc used sparingly USUALLY 

13. When ranking, the number of items to be ranked is 

limited (e.g., three best and three worst) USUALLY 

14. For checklists: 

a. If I % a line is skipped after every three to 

i^^"^* USUALLY 

b . Column headings arc carried over ftom one page 

USUALLY 

c. Column headings arc presented parallel, xather 

than perpendicular, to the iiei:. stem USUALLY 



SOMETIMES SELDOM 



SOMETIMES 
SOMETIMES 

SOMETIMES 
SOMETIMES 
SOMETIMES 



SOMETIMES 
SOMETIMES 

SOMETIMES 
SOMETIMES 
SOMETIMES 



SOMETIMES 
SOMETIMES 
SOMETIMES 



SELDOM 
SELDOM 

SELDOM 
SELDOM 
SELDOM 



SOMETIMES SELDOM 



SELDOM 
SELDOM 

SELDOM 
SELDOM 
SELDOM 



SOMETIMES SELDOM 



SELDOM 
SELDOM 
SELDOM 



(Please continue to page 8) 
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Demographic Information 

PiMse provide answcn to the foUowing questions KganJing your background and experience in survey research 
1 . How many years have you been involved in survey research activities? 



2. How many jnail surveys were you involved in conducting during 1 988? 

3. Would you consider 1988 a tyjrical yearin regard to your survey activities? 
Yes 

r«u, less than usual 

No, more than usual 

4. Have you ever conducted and reported any studies of survey or questionnaire methodology to find more effective 
ways of conducting surveys? 

Yes 

No 

^ ^J^^^^ V^Jblishcd an article or book on survey methodology or results of your research regarding survey 

Yes 

No 

6. What type of mail surveys are you generally involved in? 

^Cons umcr resccrch 

Public opinion polls 

Insdtutional/organizalion research 

Other 



7. What typc($) of population do you generally survey? 
G eneral puMic 

Alumni 

Program participants 
Users of a particular product 

Members of an organization or specific group, such as employees 

Other. 



8. What is the nature of your employment? 

CollegeAJniversily faculty 

Research Institute in a college/university 

Employed by large organization as part of research division or section 

Survey consultant (private enterprise) 

Other 



Thank you for sharing yoMr experience and expertise with us in this research. 

Please return to Judy Boser, The University of Tennessee, 212 Claxton, 
Knoxviilei TN 37996» 
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